fix(events): repair torn outbox lines and report the ones that cannot be read - #900

What

Fix 4 of the torn-line sweep, on the event outbox. Two parts, and they are separate claims.

Terminator repair. emit() appended in text mode and wrote json + "\n". A previous write that died between write() and fsync() leaves a last line with no terminator, so the next append glues two records into one line. The damage runs FORWARD from the tear: the torn record and the next healthy append fuse into one unparseable line, while the record before the tear is untouched. So a torn write does not cost one event, it costs that record and the next one written after it — permanently, because every later append builds on the glued line. What the repair buys is precise and worth stating exactly: it cannot recover a record that was never fully written, and it saves the next one. Without it you lose two; with it you lose the one the crash truncated. emit() now probes the last byte under the existing write lock and heals the seam before writing.

The count. Both readers skipped unusable lines in silence. For the channel bridge an unreadable line is a message that never reaches a channel. It is reported on every read, not once: the cursor filter applies only to lines that parse, so a line with no usable seq can never be advanced past, wherever it sits in the file — position is irrelevant, and mid-file lines repeat exactly as trailing ones do. That is deliberate rather than incidental — the line is still there and still unreadable — and how long it stays that way depends on which kind of tear produced it, a distinction the word "torn" hides:

So only a truncated record is permanently unreadable, and an outbox carrying one reports it on every read until someone repairs the file. Listed as a residual below. Both rows are pinned by witnesses, and they are each other's control. read_events_detailed() returns the events and the lines it could not read, from the same read — a second pass would describe a different moment, and the outbox is appended to concurrently. read_events() stays list-shaped for the eleven call sites that index and len() it — all of them tests, and zero production callers remain once the bridge moves to read_events_detailed(), so the wrapper is a test-compatibility surface rather than a load-bearing API. An earlier version of this description said seventeen; that was an artifact of a substring query counting a def, two prose mentions inside strings, and four hits on an unrelated _read_events helper in another test file. The bridge reports on stderr so stdout stays parseable.

The count is reported from the read path only. _current_seq() runs once per emit inside the write lock, where a count would be per-append and aimed at whoever happened to be writing. Its docstring states that, because an omission and a decision look identical in code.

Also guarded, each with its own witness:

• the decode moved to bytes-per-line, so a single invalid byte can no longer escape as UnicodeDecodeError from outside every guard;
• the parse guard's exception tuple is now derived from what json.loads can raise rather than from what had been seen: ValueError (the base class of JSONDecodeError) and RecursionError, which is not a ValueError, so deep nesting used to escape the old (JSONDecodeError, TypeError) pair. Structure is checked separately, since a valid JSON array parses and is still not an event;
• seq values are type-validated: bool is an int subclass, and a float becomes inf and serializes as Infinity, which is not valid JSON;
• a corrupt cursor no longer bricks reads.

Three things this fix got wrong first

None of them were found by its own witnesses, which is the part worth recording.

An early version swallowed OSError in the line iterator. That is precisely the OSError-to-zero fallback an earlier fix removed on purpose: swallow it and _current_seq() returns 0, and emit() then allocates sequence numbers that duplicate live ones. A pre-existing test stood guard over that contract and caught it. Content errors are data and get skipped-and-counted; an I/O error is not knowing what the file holds and must fail closed. The reader tolerates FileNotFoundError only — rotation renames the file — and propagates every other OSError, because a reader that swallows EIO reports "no new events" forever.

MalformedLine and EventRead were dataclasses. This module is loaded out-of-tree by spawned workers via spec_from_file_location() + exec_module(), which does not register it in sys.modules, and @dataclass resolves field types through exactly that. Every worker died at import, and the symptom was a concurrency test timing out with "neither writer reached sequence allocation" — a failure that reads like flakiness. Now plain classes, with the constraint written into the file and a witness that fails in 0.06 s naming the AttributeError instead of after a 10 s timeout.

The default report echoed raw line content to stderr, verbatim, including an API-key-shaped string in a measured probe. stderr is copied into CI logs, scrollback, and pasted transcripts. Redacting was rejected: matching secret-shaped patterns in arbitrary bytes is a denylist over untrusted input, the same defect shape the parse guard exists to avoid. The excerpt stays on the data object for callers that need it; the default report prints ordinal and reason, both structural, with show_content=True as an opt-in.

One existing test's monkeypatch target moved from read_text to read_bytes because the read verb changed. Its contract and every assertion are unchanged; without the move it would have gone green by missing its target rather than by the behaviour holding. Both it and the new witness now undo the patch before reading the file back, since a verification must not travel the path the test deliberately sabotaged.

Two statements in an earlier version of this description were wrong, and a reviewer measured both: the glue direction was stated backwards, and the unreadable-line count was claimed to be reported once. The whole suite was green while the prose contradicted the measured behaviour — the prose and the code comments agreed with each other, which is why nothing looked wrong; what neither matched was what the code did. Nothing pinned it. Three witnesses now do, including the mid-file case, which shows the rule is broader than the trailing-line framing that surfaced it.

Correcting this description and the commit message was not enough, and a second review caught that: the same backwards claim was still in the production repair comment and the test module docstring. Fixing the cited surfaces and stopping there left the code asserting one direction while this description asserted the other — an artifact contradicting itself, which is worse than being uniformly wrong. Running a query for the whole class rather than the cited instances then found a third live instance neither review named: the reported-once claim, still standing in that same docstring. All are corrected, the class query comes back clear, and the executable syntax tree is unchanged by those edits.

A sixth review found three more, and all three were prose — the code it gated was clean. The one worth the round: the reported-once claim was still live in the channel-bridge comment, in the bridge hunk of all five versions, and every class query I ran missed it because it paraphrases the sentence the earlier reviews cited rather than repeating it. A query built from a cited instance finds copies, not paraphrases; the class is the claim, and the only instrument that finds a paraphrase is reading the hunk. The other two are the two corrections above — the parse-guard framing, and the call-site count, which both reviewers had co-signed at v1 from that same substring query, so it needed re-deriving rather than re-reading. Sweeping the count as a class then turned up three further copies in docstrings that no review had cited.

A seventh review found two more, both measured, and one of them corrected a claim in the production code rather than in this text. F4: the replacement bridge comment said an unterminated record is reported and then heals at the next emit(). Measured false. An unterminated record whose bytes are complete is never reported at all — _iter_outbox() splits on b"\n", so a complete final chunk parses on the spot, before any emit, repair or no repair. The sentence implied a window of unreadability that does not exist. It is recut, and the gap that let it survive is now pinned: every tear fixture in this file emitted after tearing, so nothing had ever asked what the reader alone does with a torn file. W11 is that pair — the unterminated case reporting nothing across two reads, and the truncated case reporting one as its discriminating control. The reviewer's probe is the witness; I took it rather than restating it. The phrase's two siblings, the W10 header and the table above, are scoped to tear -> emit -> read and are true there, so they are deliberately unchanged.

F5: the suite totals were stale, corrected above. The mechanism is worth stating because the reviewer's diagnosis and my measurement differ: the +3 is not the #899 merge — that added 21 tests, and every one of them is inside both baselines — it is gr2/gr2_overlay/tests/, which the narrower invocation does not collect. Both figures were correct measurements of different scopes; the defect was publishing one without naming which. Chasing it down is what surfaced the three stale mutation rows, which had gone out of date the same way and which no review had asked about.

Evidence (RAN)

• 37 witnesses, none before. Terminator repair including a second tear and a fresh-file case; undecodable bytes; eight hostile-but-valid content rows; the I/O-versus-content boundary with a discriminating control; the count reaching a consumer, with a negative case; the out-of-tree loader constraint; three pinning how long an unreadable line keeps being reported and which record the glue destroys; and three for the truncated tear, added after a reviewer found a claim that a torn line self-heals at the next emit — TRUE of an unterminated record and FALSE of a truncated one, which is the whole problem with the bare word. That prose was wrong because every tear fixture until then dropped the trailing newline while leaving the record complete — the lucky case, and the only one I had ever tested. The realistic tear truncates the record mid-JSON, and the new witnesses carry a discriminating control — the unterminated tear, which DOES fully recover — so the finding is not mistaken for the norm. That control then caught the overcorrection: a later draft of this description swung to "a torn record never becomes readable," which the control disproves. The word "torn" covers two cases with opposite outcomes; this text now names them separately and never uses the bare word to make a claim about recovery. W11 is a later pair, reading a torn file with NO intervening emit, and it pins the sharper fact a seventh review measured: an unterminated record is never reported in the first place — not that it recovers — with the truncated case as its discriminating control.
• Nine mutation rows, re-measured for this version over the two events spec files (71 tests) — the scope is stated because omitting it is exactly what made three of them wrong: text-mode decode → real UnicodeDecodeError (11); the old exception denylist → real RecursionError (1); terminator repair removed → 5; seq validation removed → TypeError (4); count silenced → 19; reporter silenced → exactly 1, which is the discrimination proving the count reaches a consumer rather than sitting in a field; OSError swallowed → 3; FileNotFoundError widened to OSError → 1; cursor guard removed → real TypeError (1). Restores hash-verified after every row, with the unmutated pair re-run as a control. Three of these rows were stale in v7 — published as 8, 3 and 12 where the true figures are 11, 5 and 19. They were measured early and never re-derived while five review rounds added witnesses to the very files they count. The code never changed; the evidence describing it went out of date silently, which is the same defect as a stale suite total and is invisible to every check in this PR except re-running it.
• Full suite at both scopes, base pinned by SHA — omitting the scope is what made v7's totals unverifiable. On origin/dev@936756770e5520d222dfbbe1a34f366094c28f0d in an isolated worktree: pytest gr2/tests 1032 passed / 5 failed, pytest gr2 1035 / 5. On this head: 1069 / 5 and 1072 / 5. Delta exactly +37 in both scopes, matching the 37 witnesses, failure sets identical in both directions. The two scopes differ by gr2/gr2_overlay/tests/test_overlay_refs_namespace.py — 3 tests, measured passing on both sides, which pytest gr2/tests does not collect. v7 published 1067 / 1032: right for the narrow scope, and unverifiable to anyone running the wider one.
• Every claim this description makes about behaviour is pinned by a named witness — 18 audited, 0 unpinned: the seventeen from v7 re-checked, plus the new an unterminated record is never reported claim, pinned by W11. Stated with its limit, because the check is weaker than it sounds: it verifies that a witness with the matching name exists and passes, not that the witness asserts exactly the sentence above it. It narrows the gap between prose and behaviour; it does not close it. Seven separate reviews found wrong behavioural claims in earlier versions of this text, every one of them surviving a green suite, so the audit exists because that class of defect is invisible to every other check here.
• Lint unchanged everywhere, each measured in place with a swap-confirmed control: production files 6 = 6, the edited test file 23 = 23, the new file 0.

Residuals

Named rather than presented as clean:

• These primitives now exist in three places. Consolidating them is outside this fix's scope: this file is production and the other copies are prototypes, so the dependency would run the wrong way.
• The same content-echo property exists in the prototype reporter merged earlier and is not addressed here.
• Unbounded line length remains a read-layer resource limit and is deliberately undefended.
• A truncated record — an unterminated one is readable throughout and never reaches the report at all — is unreadable permanently: no later emit() repairs it, because the repair heals the seam and not the record, so every read reports it and a consumer polling in a loop warns every cycle until someone repairs the file. Suppressing that would need persisted already-reported state and would hide a re-occurring fault; reporting it is the lesser cost, but on a damaged outbox it is a standing cost, not a transient one.

Premium boundary: grip is OSS; this is local file mechanics over opaque paths and carries no identity, org, or policy content.